1. Field of the Invention
The present invention relates to an encoder and a decoder capable of eliminating block distortion of an image.
2. Description of the Background Art
An encoder for compressing an original image and a decoder for expanding the compressed image reduce the data amount of images to be transmitted in a communications system. One such conventional encoder and decoder is disclosed in “HD Photo-Photographic Still Image File Format”, [online], Nov. 7, 2006, Microsoft Corporation, [retrieved on Oct. 10, 2007]. Retrieved from the Internet: <URL:http://www.microsoft.com/whdc/xps/hdphotodpk.mspx>
{The Encoder According to the Background Art}
The encoder disclosed in “HD Photo-Photographic Still Image File Format” is described as follows with reference to FIGS. 13 and 14. The encoder 1P includes a first hierarchical layer 11P, a second hierarchical layer 12P, and a compressor 13P. The first hierarchical layer 11P includes an overlap processor 111P and a frequency conversion section 112P. The second hierarchical layer 12P includes an overlap processor 121P and a frequency conversion section 122P.
On the first hierarchical layer 11P, an original image is inputted from outside the encoder 1P, and then frequency-converted by the frequency conversion section 112P. Then, the DC components of the first hierarchical layer 11P are outputted to the second hierarchical layer 12P and the AC components of the first hierarchical layer 11P are outputted to the compressor 13P.
On the second hierarchical layer 12P, the DC components on the first hierarchical layer 11P are inputted therefrom, and then frequency-converted by the frequency conversion section 122P. The DC and AC components on the second hierarchical layer 12P are outputted to the compressor 13P.
The compressor 13P inputs the AC components on the first hierarchical layer 11P therefrom and the DC and AC components on the second hierarchical layer 12P therefrom. The compressor 13P then compresses data of each component and outputs a compressed image outside the encoder 1P. Thus, the encoder 1P provides efficient data compression.
On the first hierarchical layer 11P, the overlap processor 111P performs overlap processing before the frequency conversion section 112P performs frequency conversion. On the second hierarchical layer 12P, the overlap processor 121P performs overlap processing before the frequency conversion section 122P performs frequency conversion. The overlap processing is performed across four adjacent block areas. As a result, block distortion of an image is eliminated.
Actually, the user can configure the overlap processor 111P or 121P not to operate. For example, the user can configure that the overlap processor 111P on the first hierarchical layer 11P operates, but the overlap processor 121P on the second hierarchical layer 12P does not.
The following is a description of the hierarchical processing by the encoder 1P when the encoder 1P inputs an original image shown in FIG. 14A. An input image A1 on the first hierarchical layer 1P has 16 block areas of 4×4, including block areas A11, A12, A13, and A14. In the following description, a P×Q block area has P pixels in the horizontal direction and Q pixels in the vertical direction.
In an overlap processing image A2 on the first hierarchical layer 11P shown in FIG. 14B, overlap processing is performed across four adjacent block areas, as shown by dotted lines. For example, a 4×4 overlap processing area A24 equally straddles block areas A11, A12, A13, and A14.
At the boundaries of the overlap processing image A2 on the first hierarchical layer 11P, an overlap processing area of 4×4 is not secured, but overlap processing areas of 2×2, 4×2, and 2×4 are secured. For example, overlap processing areas A21 of 2×2, A22 of 4×2, and A23 of 2×4 are secured.
An output image A3 on the first hierarchical layer 11P shown in FIG. 14C has 16 DC components including DC components A31, A32, A33, and A34, and also AC components.
An input image A4 on the second hierarchical layer 12P shown in FIG. 14D has a 4×4 block area A41. The block area A41 has the 16 DC components of the output image A3 on the first hierarchical layer 11P.
In an overlap processing image A5 on the second hierarchical layer 12P shown in FIG. 14E, overlap processing areas A51, A52, A53, and A54 of 2×2 shown by dotted lines are secured.
An output image A6 on the second hierarchical layer 12P shown in FIG. 14F has a DC component A61 and AC components.
{The Decoder According to the Background Art}
The decoder disclosed in “HD Photo-Photographic Still Image File Format” is described as follows with reference to FIGS. 15 and 16. The decoder 2P includes an expander 21P, a second hierarchical layer 22P, and a first hierarchical layer 23P. The second hierarchical layer 22P includes a frequency inversion section 221P and an overlap inverse processor 222P. The first hierarchical layer 23P includes a frequency inversion section 231P and an overlap inverse processor 232P.
The expander 21P inputs a compressed image from outside the decoder 2P and expands data of each component. The expander 21P then outputs the DC and AC components on the second hierarchical layer 22P thereto and further outputs the AC components on the first hierarchical layer 23P thereto.
On the second hierarchical layer 22P, the DC and AC components on its own hierarchical layer are inputted from the expander 21P and then frequency-inverted by the frequency inversion section 221P. The DC components on the first hierarchical layer 23P are outputted thereto from the second hierarchical layer 22P.
On the first hierarchical layer 23P, the DC components on their own hierarchical layer are inputted from the second hierarchical layer 22P and the AC components on their own hierarchical layer are inputted from the expander 21P, and then frequency-inverted by the frequency inversion section 231P. Then, an expanded image is outputted from the first hierarchical layer 23P outside the decoder 2P. Thus, the decoder 2P provides efficient data expansion.
On the second hierarchical layer 22P, the overlap inverse processor 222P performs overlap inverse processing after the frequency inversion section 221P performs frequency inversion. On the first hierarchical layer 23P, the overlap inverse processor 232P performs overlap inverse processing after the frequency inversion section 231P performs frequency inversion. The overlap inverse processing is performed across four adjacent block areas. As a result, block distortion of an image is eliminated.
Actually, the user can configure the overlap inverse processor 222P or 232P not to operate. For example, the user can configure that the overlap inverse processor 232P on the first hierarchical layer 23P operates, but the overlap inverse processor 222P on the second hierarchical layer 22P does not.
The following is a description of the hierarchical processing by the decoder 2P when the decoder 2P outputs an expanded image shown in FIG. 16F. An input image B1 on the second hierarchical layer 22P shown in FIG. 16A has a DC component B11 and AC components.
In an overlap inverse processing image B2 on the second hierarchical layer 22P shown in FIG. 16B, overlap inverse processing areas B21, B22, B23, and B24 of 2×2 shown by dotted lines are secured.
An output image B3 on the second hierarchical layer 22P shown in FIG. 16C has a 4×4 block area B31. The block area B31 has the 16 DC components of the input image B4 on the first hierarchical layer 23P.
An input image B4 on the first hierarchical layer 23P shown in FIG. 16D has 16 DC components including DC components B41, B42, B43, and B44 and also AC components.
In an overlap inverse processing image B5 on the first hierarchical layer 23P shown in FIG. 16E, overlap inverse processing is performed across four adjacent block areas, as shown by dotted lines. For example, a 4×4 overlap inverse processing area B54 equally straddles block areas B61, B62, B63, and B64.
At the boundaries of the overlap inverse processing image B5 on the first hierarchical layer 23P, an overlap inverse processing area of 4×4 is not secured, but overlap inverse processing areas of 2×2, 4×2, and 2×4 are secured. For example, overlap inverse processing areas B51 of 2×2, B52 of 4×2, and B53 of 2×4 are secured.
An output image B6 on the first hierarchical layer 23P has 16 block areas of 4×4 including the block areas B61, B62, B63, and B64.
{The Problems of the Encoder to be Solved}
The following is a description of the problems of the encoder 1P to be solved. The frequency conversion section 112P cannot complete frequency conversion of the overlap processing image A2 on the first hierarchical layer 11P, until the overlap processor 111P completes overlap processing on the input image A1 on the first hierarchical layer 11P. The frequency conversion section 122P cannot complete frequency conversion of the overlap processing image A5 on the second hierarchical layer 12P, until the overlap processor 121P completes overlap processing on the input image A4 on the second hierarchical layer 12P.
The original image that the encoder 1P actually inputs generally has a large number of 16×16 image areas. The overlap processor 111P is required to hold the data of a 16×16 image area out of the original image until the image area undergoes the overlap processing. The overlap processor 121P has the same problem. This lowers the performance of the encoder 1P, thus reducing the advantage of the high performance to be achieved by hardware implementation.
The following is a description, with reference to FIGS. 17A to 17F and 18, of how much data the overlap processor 111P is required to hold. In FIGS. 17A to 17F and 18, for simplification, the following description assumes a case where the higher, first hierarchical layer 11P performs overlap processing, but the lower, second hierarchical layer 12P does not. Since block distortion appears along block boundaries, this assumption is practical. The following description also assumes that the encoder 1P inputs a 32×48 original image which has two 16×16 image areas in the horizontal direction and three 16×16 image areas in the vertical direction.
As shown in FIG. 17A, the overlap processor 111P inputs the first 16×16 image area and performs overlap processing on a 14×14 image area C11, but not on the remaining image area C12. The overlap processor 111P performs overlap processing on the last image area after inputting image areas adjacent thereto.
As shown in FIG. 17B, the overlap processor 111P inputs the second 16×16 image area and performs overlap processing on a 2×14 image area C121 of the image area C12, and a 16×14 image area C21, but not on a 16×2 image area C122 of the image area C12, and a 16×2 image area C22. The overlap processor 111P performs overlap processing on the last two image areas after inputting image areas adjacent thereto.
As shown in FIG. 17C, the overlap processor 111P inputs the third 16×16 image area and performs overlap processing on a 14×2 image area C1221 of the image area C122, and a 14×14 image area C31, but not on a 2×2 image area C1222 of the image area C122, the image area C22, and the remaining image area C32. The overlap processor 111P performs overlap processing on the last three image areas after inputting image areas adjacent thereto.
As shown in FIG. 17D, the overlap processor 111P inputs the fourth 16×16 image area and performs overlap processing on the image area C1222, the image area C22, a 2×14 image area C321 of the image area C32, and a 16×14 image area C41, but not on a 16×2 image area C322 of the image area C32, and a 16×2 image area C42. The overlap processor 111P performs overlap processing on the last two image areas after inputting image areas adjacent thereto.
Thus completing the overlap processing on the first and second inputted 16×16 image areas, the overlap processor 111P does not need to hold the data of these image areas any more. The frequency conversion section 112P performs frequency conversion on these image areas.
As shown in FIG. 17E, the overlap processor 111P inputs the fifth 16×16 image area and performs overlap processing on a 14×2 image area C3221 of the image area C322, and a 14×16 image area C51, but not on a 2×2 image area C3222 of the image area C322, the image area C42, and a 2×16 image area C52. The overlap processor 111P performs overlap processing on the last three image areas after inputting an image area adjacent thereto.
As shown in FIG. 17F, the overlap processor 111P inputs the sixth 16×16 image area and performs overlap processing on the image areas C3222, C42, C52, and a 16×16 image area C6.
Thus completing the overlap processing on the third to sixth inputted 16×16 image areas, the overlap processor 111P does not need to hold the data of these image areas any more. The frequency conversion section 112P performs frequency conversion on these image areas.
The following description assumes a general case where the original image includes N image areas of 16×16 in the horizontal direction. After the first to N-th inputs, the overlap processor 111P has not completed overlap processing on the 16×16 image areas inputted so far.
Therefore, as shown in FIG. 18, the overlap processor 111P is required to hold 16×16 image data after the first input, 32×16 image data after the second input, and 16N×16 image data after the N-th input.
After the (N+1)th input, the overlap processor 111P has not completed overlap processing on the 16×16 image areas inputted so far. After the (N+2)th input, however, the overlap processor 111P completes overlap processing on the first inputted 16×16 image area. Similarly, after the (N+3)th input, the overlap processor 111P completes overlap processing on the second inputted 16×16 image area.
Therefore, as shown in FIG. 18, the overlap processor 111P is required to hold up to 16(N+1)×16 image data. More specifically, the overlap processor 111P needs to hold image data of approximately up to the product of (the vertical width of the input unit area)×(the horizontal width of the original image). The term “input unit area” means a 16×16 image area that the encoder 1P inputs at one time.
{The Problems of the Decoder to be Solved}
The following is a description of the problems of the decoder 2P to be solved. The frequency inversion section 231P cannot perform frequency inversion of the input image B4 on the first hierarchical layer 23P, until the overlap inverse processor 222P completes overlap inverse processing on the overlap inverse processing image B2 on the second hierarchical layer 22P. The decoder 2P cannot output the output image B6 on the first hierarchical layer 23P, until the overlap inverse processor 232P completes overlap inverse processing on the overlap inverse processing image B5 on the first hierarchical layer 23P.
The expanded image that the decoder 2P actually outputs generally has a large number of 16×16 image areas. The overlap inverse processor 232P is required to hold the data of a 16×16 image area out of the expanded image until the image area undergoes the overlap inverse processing. The overlap inverse processor 222P has the same problem. This lowers the performance of the decoder 2P, thus reducing the advantage of the high performance to be achieved by hardware implementation.
The following is a description, with reference to FIGS. 17A to 17F and 18, of how much data the overlap inverse processor 232P is required to hold. In FIGS. 17A to 17F and 18, for simplification, the following description assumes a case where the higher, first hierarchical layer 23P performs overlap inverse processing, but the lower, second hierarchical layer 22P does not. Since block distortion appears along block boundaries, this assumption is practical. The following description also assumes that the decoder 2P outputs a 32×48 expanded image which has two 16×16 image areas in the horizontal direction and three 16×16 image areas in the vertical direction.
As shown in FIG. 17A, the overlap inverse processor 232P inputs the first 16×16 image area and performs overlap inverse processing on a 14×14 image area C11, but not on the remaining image area C12. The overlap inverse processor 232P performs overlap inverse processing on the last image area after inputting image areas adjacent thereto.
As shown in FIG. 17B, the overlap inverse processor 232P inputs the second 16×16 image area and performs overlap inverse processing on a 2×14 image area C121 of the image area C12, and a 16×14 image area C21, but not on a 16×2 image area C122 of the image area C12, and a 16×2 image area C22. The overlap inverse processor 232P performs overlap inverse processing on the last two image areas after inputting image areas adjacent thereto.
As shown in FIG. 17C, the overlap inverse processor 232P inputs the third 16×16 image area and performs overlap inverse processing on a 14×2 image area C1221 of the image area C122, and a 14×14 image area C31, but not on a 2×2 image area C1222 of the image area C122, the image area C22, and the remaining image area C32. The overlap inverse processor 232P performs overlap inverse processing on the last three image areas after inputting image areas adjacent thereto.
As shown in FIG. 17D, the overlap inverse processor 232P inputs the fourth 16×16 image area and performs overlap inverse processing on the image area C1222, the image area C22, and a 2×14 image area C321 of the image area C32, and a 16×14 image area C41, but not on a 16×2 image area C322 of the image area C32, and a 16×2 image area C42. The overlap inverse processor 232P performs overlap inverse processing on the last two image areas after inputting image areas adjacent thereto.
Thus completing the overlap inverse processing on the first and second inputted 16×16 image areas, the overlap inverse processor 232P does not need to hold the data of these image areas any more. The decoder 2P outputs the expanded image of these image areas.
As shown in FIG. 17E, the overlap inverse processor 232P inputs the fifth 16×16 image area and performs overlap inverse processing on a 14×2 image area C3221 of the image area C322, and a 14×16 image area C51, but not on a 2×2 image area C3222 of the image area C322, the image area C42, and a 2×16 image area C52. The overlap inverse processor 232P performs overlap inverse processing on the last three image areas after inputting an image area adjacent thereto.
As shown in FIG. 17F, the overlap inverse processor 232P inputs the sixth 16×16 image area and performs overlap inverse processing on the image area C3222, C42, C52, and a 16×16 image area C6.
Thus completing the overlap inverse processing on the third to sixth inputted 16×16 image areas, the overlap inverse processor 232P does not need to hold the data of these image areas any more. The decoder 2P outputs the expanded image of these image areas.
The following description assumes a general case where the expanded image includes N image areas of 16×16 in the horizontal direction. After the first to N-th inputs, the overlap inverse processor 232P has not completed overlap inverse processing on the 16×16 image areas inputted so far.
Therefore, as shown in FIG. 18, the overlap inverse processor 232P is required to hold 16×16 image data after the first input, 32×16 image data after the second input, and 16N×16 image data after the N-th input.
After the (N+1)th input, the overlap inverse processor 232P has not completed overlap inverse processing on the 16×16 image areas inputted so far. After the (N+2)th input, however, the overlap inverse processor 232P completes overlap inverse processing on the first inputted 16×16 image area. Similarly, after the (N+3)th input, the overlap inverse processor 232P completes overlap inverse processing on the second inputted 16×16 image area.
Therefore, as shown in FIG. 18, the overlap inverse processor 232P is required to hold up to 16(N+1)×16 image data. More specifically, the overlap inverse processor 232P needs to hold image data of approximately up to the product of (the vertical width of the output unit area)×(the horizontal width of the expanded image). The term “output unit area” means a 16×16 image area that the decoder 2P outputs at one time.